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METHOD AND SYSTEM FOR INTERACTIVELY DEVELOPING A GRAPHICAL 
CONTROL-FLOW AND ASSOCIATED APPLICATION SOFTWARE FOR MACHINE 
VISION SYSTEM 



5 Technical Field 

This invention relates to methods and systems 
for interactively developing a graphical control -flow 
structure and associated application software for use in 
a machine vision system. 

io Background Art 

Traditionally, vision tools are programmed 
together within a programming environment such as C++, 
Delphi, or Visual Basic wherein a set of tools can be 
created, used and configured together. Although some 
15 environments are easier to program than others (e.g. 

Visual Basic is easier than C++) , they still require 
knowledge of that programming environment . 

The U.S. patent to Silver et al. (5,481,712) 
discloses a method and system for interactively generat- 

2 0 ing a computer program in "C" for machine vision analy- 
sis wherein the program only permits the operator (i.e. 
programmer) to make only syntactically correct modifica- 
tions to the program. A menu element displays permissi- 
ble programming modifications for the location of 

25 interest. The menu element incorporates in its display 
of permissible programming modifications statements for 
machine vision .analysis of an object image such as calls 
to machine vision, sub-routines and functions. The 
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computer program is programmed via a closed or propri- 
etary package via predefined tools and operations. 

The U.S. patent to Lipkis et al . (4,914,567) 
provides for a design system using visual language. The 
5 system uses a visual user presentation of a control 
system, including structure, specification and opera- 
tion. The visual element enables a user to interactive- 
ly design, modify and explore operating characteristics. 
A Graf cet -based graphic control -flow language for 
10 setting and testing state variables is disclosed. 

Additional features are: application-artificial vision 
systems; object-oriented applications platform; top-down 
design; and setting of variable values. The system 
links Grafcet and MAP technology. 

15 The U.S. patent to Rumbaugh et al . (5,0 05,119) 

provides for user interactive control of computer 
programs and corresponding versions of input/output data 
flow. Programs for a CAD task and their respective data 
set requirements are visually displayed as a flowgraph 

20 with which the user interacts to select input data sets 
and initiate program executions. 

The U.S. patent to Endo (5,327,350) provides 
for an interactive- type data input function for control- 
ling the operation of a machine tool by creating a 
25 machining program based on input data. 

The U.S. patent to Fujita et al . (5,3 83,110) 
adds the feature of answering questions displayed on a 
display screen to generate a numerical control program. 

The U.S. patent to Kahn (5,157,663) provides 
3 0 for an animated user interface for computer program 
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creation, control and execution. The interface enables 
the user to create programs in an interactive visual 
manner. Animation is integrated with computer program- 
ming to allow a user to visualize programming flow and 
operation . 

The U.S. patent to Lane et al . (4,679,137) 
provides for a process control interface system for 
designer and operator. A dynamic menu element is used 
during the definition and selection of operating parame- 
ters used by a process control program. Each data 
structure includes value indicia defining the parameters 
of the data structures. 

The U.S. patent to Iadipaolo et al . 
(4,759,074) provides for a method for automatically 
inspecting parts utilizing machine vision and system 
utilizing same. The system is taught by a series of 
steps which features are to be inspected. Typical 
features, as well as tolerances, are defined by the 
programmer to develop reference data. 

The U.S. patents to Hayward et al . 
(5,574,828), Abdel-Malek (5,511,147), Kuwahara et al . 
(4,833,624), Wright et al . (5,453,933), Spight 
(4,462,046), Gordon (5,314,055), each provide general 

teachings of user interactive programming techniques for 

robotic or CNC machine tool environments. 

In the automation world (i.e. for PLC's), 
there exists open standards such as Grafcet and IEC 1131 
to define the syntax for automation control. 
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Summarv Of The Invention 

An object of the present invention is to 
provide a method and system for interactively developing 
application software for use in a machine vision system 
5 wherein the user teaches an imaging programming task 
without writing any code. Consequently, the user need 
not be a programmer. Typically, a programmed computer 
queries the user's intentions and guides the user's 
choices until the task is completely taught. In the 
10 process, the user does not write a single line of code, 
but rather sets variables that the machine vision tools 
require interactively. Certain tool's outputs typically 
will be another tool's input. 

Another object of the present invention is to 
15 provide a method and system for interactively developing 
application software for use in a machine vision system 
wherein the application software may be developed in a 
hierarchy which allows bottom-up or top-down program 
development, thereby adapting to the user's thinking. 

2 0 For example, to measure a distance a ruler tool requires 

two input geometries and, in turn, outputs a distance. 
An edge tool may define an edge and feature find tools 
match a model to define a point. Both these can be the 
inputs to the ruler tool. The user may start top-down, 
25 i.e. he thinks measurement first and the system guides 
him to define the input geometries. Alternatively, the 
user may think bottom-up and define the geometries first 
and then decide to do a measurement between them. 

Yet another object of the present invention is 

3 0 to provide a method and system for interactively devel- 

oping application software for use in a. machine vision 
system wherein the method and system allow the easy 
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integration of third party tools (by using standard 
controls, such as ActiveX controls) . Not only can the 
standard properties of the controls be inputs or outputs 
to other tools, but also objects such as geometries can 
5 be used as inputs and outputs to other tools. 

Yet still another object of the present 
invention is to provide a method and system for interac- 
tively developing application software for use in a 
machine vision system wherein, to represent the flow 
10 through the taught task, a modified Grafcet standard is 
utilized to accept standard controls and objects as part 
of the steps and transitions. In the Grafcet standard, 
transitions can access any control ' s properties or 
objects to make execution judgments. 

15 In carrying out the above objects and other 

objects of the present invention, a method is provided 
for developing a graphical, control- flow structure and 
associated application software for use in a machine 
vision system utilizing a computer system. The struc- 

20 ture includes a control sequence having at least one 
node, at least one transition, and at least one link for 
interconnecting the at least one node and the at least 
one transition. The method includes the step of storing 
an application development program, including a first 

25 set of control programs representing possible components 
of a user interface for the machine vision system. The 
first set of control programs define a first set of 
standard controls. The method also includes the step of 
storing a second set of control programs representing 

3 0 possible machine vision algorithms for the machine 
vision system. The second set of control programs 
define a second set of standard controls. The method 
further includes the step of storing hardware operating 
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parameters corresponding to possible hardware for use in 
the machine vision system. The hardware operating 
parameters define a third set of standard controls. The 
method further includes the step of displaying graphical 
5 representations of the possible components, the possible 
hardware and the possible machine vision algorithms. 
Then, the method includes the step of receiving commands 
from a user of the computer system to select a first 
control program corresponding to a desired component of 
10 the user interface, desired hardware operating parame- 
ters corresponding to desired hardware and a machine 
vision graphical representation and its associated 
second control program corresponding to a desired 
machine vision algorithm. The method also includes the 
15 step of displaying the structure. The graphical repre- 
sentation is a node of the structure. The selected 
second control program is mapped into the structure at 
the node. The displayed structure also has at least one 
transition and at least one link for interconnecting the 
2 0 node and the at least one transition. Finally, the 
method includes the step of linking the first control 
program with the desired hardware operating parameters 
to the second control program to form the application 
software in response to the commands without the user 
25 writing any of the application software. 

Further in carrying out the above objects and 
other objects of the present invention, a system is 
provided for carrying out the method steps . 

The benefits accruing to the method and system 
3 0 of the present invention are numerous. For example, the 
method and system: 

1) interactively generate/ "teach" a computer 

program . 
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2) The marriage of two standards ActiveX (i.e. 
controls) and Grafcet and its derivative IEC 
1131, i.e. the ActiveX standard and the 
Grafcet IEC 1131 have been linked together. 
5 As a result of this, an ActiveX behavior is 

controlled via its methods. Its characteris- 
tics are controlled via setting its proper- 
ties. An ActiveX event can control the se- 
quencing in the Grafcet paradigm. 

10 3) The Grafcet/IEC 1131 standard has now been 

redefined in that a transition can be either 
True (the default case) , an event generated by 
some ActiveX control or a Boolean expression 
defined by an internal variable or properties 

15 of any ActiveX Component. 

4) A step in the Grafcet standard is now either 
any third party ActiveX control method or a 
mathematical step that allows the definition 
of new internal variables . 

2 0 5) The teach environment allows algorithms to be 

defined in a hierarchy that allows bottom-up 
or top-down teach approach. 

6) Combine vision with control. 

7) Allows the integration of any third party 
25 ActiveX controls. 

The idea behind interactively generating a 
computer program is to ensure that the programmer cannot 
make syntax errors in producing his code. Hence, the 
environment guides the programmer via menus and appro - 

3 0 priate interface structure to produce code that is 

correct in syntax. The problem being addressed is that 
computer languages typically are tightly defined and 
intolerant to syntax errors. 
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The second idea is that the syatern . operator 
does not need to be a skilled programmer. Rather, he 
can be an applications engineer who primarily under- 
stands the application and does not need to be a pro- 
5 grammer. These days, automation engineers are used to 
using programming in ladder logic and Grafcet-type 
\ languages. Applications knowledgeable engineers tend to 
\ be familiar with these environments and the interactive 
environment being presented in this application is for 
10 these engineers. 

The method and system of the present applica- 
tion is different from the examples mentioned above in 
that it takes a standard of Grafcet and modifies it to 
allow the integration of another standard, namely 
15 ActiveX controls which can be linked in. 

Hence, a first unique feature of this inven- 
tion is that any third party tools in machine vision or 
automation or otherwise that are written as ActiveX 
controls can be linked into this interactive environ- 
20 ment. To date, no interactive environment allows the 
interaction of third party components and definitely not 

ActiveX controls are becoming widely accepted 
and many are being produced by many companies. These 

2 5 are able to be linked together and used without writing 

code that can produce syntax errors . A control sequence 
is set up triggered on events, timers and variable 
settings. This interactive modified Grafcet/IEC 1131 
environment reads the properties of the ActiveX controls 

3 0 and allows these to be set or linked to other properties 

of other controls. The operator may select which 
method (s) to run on the control and events fired by 
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controls are used to sequence the logic. Hence, the 
present invention sets how any third party ActiveX 
control behaves by viewing the current standard inter- 
faces. 

5 The second unique feature * of the present 

invention is the menu structure that allows operators to 
think top-down or bottom-up in the way they program. 
Hence, we are defining a methodology analogous to the 
human operator to define an inspection routine. This is 

10 addressing the type of person who needs to use the 
system. We want the operator to have applications 
knowledge and do not want to necessitate that he also be 
a programmer. Take the following example: An operator 
wishes to measure the distance between two points. He 

15 can think metrology first and start with a rule ActiveX 
control. The interactive environment after interrogat- 
ing the ruler ActiveX control knows that two geometries 
are required. The interactive environment then queries 
all available ActiveX controls and determines which to 

2 0 present to the operator. Hence, the operator can choose 

between a line, point or other available geometry. The 
operator can select to define a new geometry or an 
existing geometry. Alternatively, the operator thinks 
points first and proceed to define the points. He then 
25 thinks metrology and selects the ruler ActiveX control 
to measure between already defined points. In this 
example, it was irrelevant whether the operator think 
geometry or metrology first. Both approaches work and 
the interactive environment create a program that runs 

3 0 and does not have any syntax errors. The whole program- 

ming sequence is done via menus and icons in a drag and 
drop environment . 



o 
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The third feature of this invention is that 
predefined interfaces are supported to exchange data 
between tools. By default ActiveX controls expose a set 
of predefined variables such as floats, double string 
5 integers, etc. The invention exposes another interface 
(COMM) which allows the user to expose his own prede- 
fined objects, and have these passed between different 
ActiveX controls. This defined interface allows for the 
management of different user defined objects between 
10 different ActiveX controls. 

The fourth characteristic of this interactive 
Graf cet /ActiveX environment is the mathematical compo- 
nent that allows the manipulation of data between 
ActiveX controls in a user friendly manner. 

15 Glossary 

ActiveX Control . A name as defined by Micro- 
soft for programmable elements formerly known variously 
as OLE Controls, OCXs , or OLE Custom Controls. It uses 
component technologies built on the Component Object 
20 Model (COM) . Generally, an ActiveX control is an object 

the functions that manipulate it. The ActiveX control 
is an entity that has state, behavior and identity. An 
object's state consists of its attributes also known as 

25 properties and the attributes' current values. An 
object's behavior consists of the operations (methods) 
that can be performed on it and the accompanying state 
changes. An object's identity is what one uses to 
distinguish it from other objects. Finally, an ActiveX 

3 0 component has events that inform when something has 
happened. In contrast, COM objects' behavior is defined 
by the interfaces it supports. A COM object's state is 



1 
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not explicitly specified, but is implied by its inter- 
faces . 

COM . An open architecture for cross-platform 
development for applications based on object oriented 
5 technology as agreed upon by Digital Equipment and 
Microsoft Corporation. The Component Object Model 
defines an interface (similar to an abstract base 
class) , Unknown, from which all COM- compatible classes 
are derived. 

10 Events . A notification message sent from an 

object in response to a state change or user action. 

Graf cet . The Grafcet is a graphical program- 
ming language - originally developed by AFCET (Associa- 
tion Francais pour la Cybernetique Economique et Tech- 

15 nique) in France and has become an international stan- 
dard. The Grafcet language represents the functions of 
an automation system as a sequence of steps and transi- 
tions - with directed flow lines connecting the two. It 
was specifically designed for sequential control prob- 

20 lems where the steps are sequential or time dependent. 

Each step in a Grafcet sequence represents a stable 
situation in an automation process and has one or more 
actions associated with it - while a transition indi- 
cates the possibility of evolution from one step to the 

25 next. Boolean events are associated with each transi- 
tion. Sequences may be linked together by a branch 
construct which may represent a conditional or a paral- 
lel flow of execution. A single Flow Line is used to 
link steps and transitions together to indicate the flow 

3 0 of execution. Double flow lines are used when two or 
more steps within a program must be synchronized. 
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IEC 1131 . A standard set down by the Interna- 
tional Electrotechnical Commission that specifies the 
syntax and semantics of a unified suite of programming 
languages for programmable controllers. Part of the 
5 standard refers to Sequential Function Charts that is 

Grafcet-like in structure. 

Methods. A procedure that provides access to 
an object's data and asks an object to behave as defined 
by the procedure . 

10 Obi ect- oriented Design . Or object-oriented 

programming. In traditional procedural languages (such 
as C, FORTRAN, and COBOL), code and data are separate. 
In the object-oriented approach, code and data that 
belong together can be combined into objects. Object- 

15 oriented design is further characterized by the use of 
inheritance (derived classes) , polymorphism, encapsula- 
tion and virtual functions (C++) in programming. 

Open . Open for our patent refers to existing 
standards that are non-proprietary and used by many 
20 companies. In the patent to Silver, open is that yes 
other people can use this system but it is a proprietary 
Cognex only standard, and not an open intercompany 
standard. Visual basic is an open programming language. 
Visual C++ is an open programming language. 

25 Properties . The attribute of an ActiveX 

control. This attribute has a value that can either be 
read, written, or both. 



30 



Transition . A Grafcet transition defines the 
state of a transition and has . an associated Boolean 
state. It indicates the possibility of evolution from 
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one step to the next. Boolean events are associated 
with each transition. 



RPL. Rapid programming language. Languages 
like Visual Basic are rapid programming languages 
5 because the programmer can quickly set up user interface 
and insert objects to achieve his goal. Visual C+ + is 
not regarded as a rapid programming language. Delphi 
and Visual .Basic are regarded as rapid programming 
languages . 

10 Step . Within the Grafcet/IEC 1131, each step 

or node in a Grafcet sequence represents a stable situa- 
tion in an automation process and has one or more 
actions associated with it. It has been redefined 
herein to represent an ActiveX or similar standard 

15 component . 

VisionBlox™ . VisionBlox™ is the tool -kit of 
machine vision components obeying the ActiveX standards. 
The components include imaging processing, image acqui- 
sition, calibration, geometric functions, and motion 
2 0 control. 



The above objects and other objects, features, 
and advantages of the present invention are readily 
apparent from the following detailed description of the 
best mode for carrying out the invention when taken in 
25 connection with the accompanying drawings. 
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Brief Description Of The Drawings 

FIGURE 1 is a schematic diagram illustrating 
a preferred hardware configuration on which the method 
of the present invention can be implemented; 

5 FIGURE 2 is a schematic diagram illustrating 

a machine vision system which can be supported by the 
method and system of the present invention; 

FIGURE 3 is a portion of a programming envi- 
ronment after standard controls of the present invention 
10 have been added; 

FIGURE 4 is schematic view of a typical 
Grafcet sequence constructed in accordance with the 
present invention; 

4 FIGURE 5 is a schematic view of a typical 

15 Grafcet sequence showing two alternative paths of 
execution; 

FIGURE 6 is a schematic view of a section of 
a Grafcet sequence showing a parallel branch; 

FIGURE 7 is a screen display illustrating 
2 0 math's control; and 

FIGURE 8 is a block diagram flow chart illus- 
trating a typical operation sequence in a normal running 
mode of the vision system which is waiting for a "part 
present" signal. 
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Best Modes For Carrying Out The Invention 

Referring now to the drawings figures, there 
is illustrated in Figure 1 a workstation on which the 
method and system of the ' present invention can be 
5 implemented. The hardware illustrated in Figure 1 

includes a monitor 10 such as a single SVGA display, a 
keyboard 12, a pointing device such a mouse 14, a 
magnetic storage device 16, and a chassis 18 including 
a CPU and random access memory. In a preferred embodi- 
10 ment, the chassis 18 is a Pentium-based IBM compatible 
PC or other PC having at least 8 megabytes of RAM and at 
least 12 megabytes of hard disk space . 

The hardware configuration also includes the 
development environment of a high-level, graphic, flow- 
15 control language such as Grafcet modified by the IEC 
1131 standard for pictorially specifying the structure 
and detailed interaction of a system of concurrent 
processes . 

Referring now to Figure 2, there is illustrat- 

2 0 ed schematically a machine vision system generally 

indicated at 20 generally of the type which can be 
supported by the method and system of the present 
invention. The machine vision system 20 typically 
includes an image digitizer/frame grabber 22. The image 
25 digitizer/frame grabber 22 samples and digitizes the 
input images from one or more image sources such as 
cameras 24 and places each input image into a frame 
buffer having picture elements. Each of the picture 
elements may consist of an 8 -bit number representing the 

3 0 brightness of that spot in the image. 
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The system 20 also includes a system bus 3 8 
which receives information from the image digi- 
tizer/frame grabber 22 and passes the information on to 
the IBM compatible host computer. 

5 The system 2 0 also includes input /output 

circuits 3 0 to allow the system 2 0 to communicate with 
external peripheral devices such as robots, programmable 
controllers, etc. having one or more stages. " 



One or more of the cameras 24 may be an image 
10 source such as an analog digital or line scan camera 
such as RS-170, CCIR, NTSC and PAL. 

The system bus 2 6 may be either a PCI an EISA, 
ISA or VL system bus or any other standard bus. 

The I/O circuits 3 0 may support a three axis 
15 stepper board (i.e. supports multiple axis control) or 
other motion boards. 

The image digitizer/frame grabber 22 may be a 
conventional frame grabber board such as that manufac- 
tured by Matrox, Cognex, Data Translation or other frame 
20 grabbers. Alternatively, the image digitizer/frame 
grabber 22 may comprise a vision processor board such as 
made by Cognex . 

The machine vision system 2 0 may be programmed 
at a mass storage unit 3 2 to include custom controls for 
25 image processing, image analysis, third party machine 
vision products, calibration, and interactive CAD/geome- 
try as described in greater detail hereinbelow. Exam- 
ples of image processing may include linear and non- 
linear enhancement, morphology, color and image arithme- 
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tic. Also, image analysis may include search, edge, 
caliper, blob, template, color, 2-D and 3-D measure- 
ments . 

Third party products may include digital I/O, 
5 various camera formats, motion, databases, SPC and 
others . 

Calibration may include non-linear, 2-D, 3-D 
and color calibration . 

Also, interactive CAD/geometry custom control 
10 may be provided for both 2-D and 3-D space. 

It is to be understood that other similar 
programming environments can be utilized by the method 
and system of the present invention. 

Referring now to Figure 3, there are illus- 
15 trated icons of a toolbox which appear, after various 
custom controls, which will be described in greater 
detail hereinbelow, are added thereto. 

Each custom control is generally described as 

follows : 

2 0 An editable shape custom control allows a user 

to define an image processing region by interactively 
editing a rectangular ellipsoidal or toroidal (donut) 
shape . 

An editable image custom control works like 
25 the visual basic picture box control with enhancements 
that give the user added image viewing and processing 
options. 
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A camera custom control allows a user to 
capture and store live images from a video camera. 

A search tool custom control looks for a 
specific feature in an image that matches a model that 
5 a user trained it to recognize; reports the characteris- 
tic of the feature found. 

A Magic custom control allow the camera 
control to capture images when the user has a Magic 
vision board installed. (This tool is invisible at run 
10 time.) 

A blob tool custpm control finds groups of 
connected pixels or "blobs" in an image; reports on the 
characteristics of each blob it found. 

An SE 100 custom control allows the camera 
15 control to capture images when the user has an SE 100 
vision board installed. (This tool is invisible at run 
time . ) 

A VP 50 custom control allows the camera 
control to capture images - and optionally speeds up 
2 0 images searches - when the user has a VP vision board 
installed. (This tool is invisible at run time.) 

A Type I stage custom control allows the stage 
control to work with a specific manufacturer's stage 
known as "Type I". (This tool is invisible at run 
2 5 time . ) 

A Stage custom control allows a user to 
control a multi-axis stage through a graphical user 
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interface. The developer can include stage control 
properties and methods in higher level code. 

While not shown, a tool board custom control 
allows a useif to navigate through windows and menus in 
5 the application by simply clicking buttons on a tool 
bar . 

Also while not shown, a Type II stage custom 
control allows the stage control to work with a specific 
manufacturer's stage known as "Type II". (This tool is 
10 also invisible at run time.) 

Finally, while also not shown, a Type III 
stage custom control allows the stage control to work 
with a specific manufacturer's stage known as "Type 
III". (This tool is furthermore invisible at run time.) 

15 In general, it is possible to interactively 

and easily build machine vision applications or programs 
using the ActiveX custom controls illustrated in Figure 
4 in the Grafcet/EIC 1131 programming system. In 
general, one creates the user interface or a vision 

2 0 application by placing the custom controls on a form. 

Next, one sets the properties for both the Visual Basic 
standard controls and the custom controls. One need not 
write code to link together the different parts of the 
application to obtain the application program since this 

25 is done automatically. 

A detailed description of many of the various 
machine vision controls represented by the icons of 
Figure 3 can be found in the above-noted patent applica- 
tion . 
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Two standards are being married together by 
the present invention to produce an open 
programmable/reconf igurable/teachable system that does 
not require the use of a software programming environ- 
5 merit . These standards are Graf cet and its derivative 
IEC 1131 and the ActiveX standard. Optionally via the 
addition of some predefined COM interfaces, the exchange 
of data between ActiveX objects is enhanced. 

Hence, the advantages of marrying these two 
10 standards allows any ActiveX components to be inserted 
within an environment that is, familiar in the automation 
(i.e. PLC) context. Secondly, by interrogating the 
properties, methods and events of these ActiveX compo- 
nents operation control can be tailored by the user 
15 without doing any programming. Properties, i.e. data, 
can be exchanged between controls. Mathematical opera- 
tion can be performed on this data. By selecting which 
method to call in a control, its operation is con- 
trolled. Finally, events control the flow of the logic. 
20 The Graf cet /IEC 1131 standard allows conditional and 
parallel control. 

This interactive Graf cet/ActiveX environment 
is itself an ActiveX component and so can be used as a 
component within another interactive Graf cet/ActiveX 

2 5 environment. 

The user configures the method and system of 
the present invention either via a drag and drop envi- 
ronment or via a menu system. The menu system is 
designed to allow the operator to define a sequence 

3 0 either bottom-up or top-down. This is achieved as the 

user, who need not be a programmer, can "teach" a task. 
The system queries his intentions, and guides the user's 
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choices until the task is taught. In the process, the 
user does not write a single line of code and has 
available to him not only all the tools, vision or 
otherwise, but can set the variables that the tools 
5 require interactively. Certain tool's outputs will be 
another tool's input. 

For example, to measure a distance a ruler 
tool requires two input geometries and, in turn, outputs 
a distance. Two other tools, e.g. an edge tool defines 
10 an edge and feature find tools match a model to define 

a point. Both these can be the inputs to the ruler 
tool . 

The user may start top-down, i.e. he thinks 
measurement first and the system guides him to define 
15 the input's geometries. Alternatively, the user may 
think bottom-up and define the geometries first and then 
decide to do a measurement between them. 

Features, Components and Modules 
Grafcet Control 

20 The Grafcet/IEC 1131 standard of the present 

invention includes transitions and steps. 

A. transition in the Grafcet/IEC 1131 standard 

is either: 

True (the default case) ; 
25 An event generated by some ActiveX Control, 

e.g. an IO ActiveX trigger control; or 

A Boolean expression defined by internal 
variable or properties of any. Control. 
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A Step in the Grafcet standard is either: 

Any 3rd party ActiveX control; or 

A mathematical step that allows the definition 
of new internal variables . 

5 Clicking with the left button of the mouse 14 

on a control gives the default control interface. 

Clicking with the right button of the mouse 14 
on the control allows the user to select which method to 
run and which properties need to be set. 

10 The Grafcet control is an ActiveX custom 

/ control that allows vision applications to be programmed 

/ in the form of "a*^5raf cet sequence or ^ slz rue t ur e . Pro- 

/ gramming takes place through a graphical user interface. 

/ Existing ActiveX controls are integrated into the 

15 Grafcet sequence. 

In one method of creating a Grafcet sequence, 
the user selects tools from a toolbar displayed on the 
Grafcet interface (such as illustrated in Figure 3) and 
drags and drops the tool onto a Sequence display area (a 

2 0 2 dimensional grid) . The steps are shown graphically 

with icons (representing the tool) which are displayed 
on the grid. Transitions and flow lines are automati- 
cally inserted to maintain a flow chart that obeys the 
rules of the Grafcet programming language (see Fig. 4) . 

25 Where inputs are required for some tools 

(steps) - the user is prompted for these through a 
message (displayed in a message panel) and the position 
for insertion of the input is highlighted. The user 
selects inputs in the same way as tools are selected - 

3 0 by dragging and dropping onto the sequence. Outputs 
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from steps will be shov/n automatically. A Grafcet 
sequence may be inserted as a step in another sequence - 
allowing for sequence reuse and multi-level sequence 
definition. The user may also insert a "parallel" or 
5 "conditional 11 branch item into the sequence (Figures 5 
and 6) . 

Within a Grafcet Control, Grafcet sequences 
may be constructed, edited and executed. A pop-up 

menu is available to access Grafcet Control member 

10 functions. To display a pop-up menu click the right 
button of the mouse 14 when the cursor is within the 
Grafcet Control Environment (NOTE: if the right mouse 
button is clicked over a Vision control - then the pop- 
up for that control will appear instead of the Grafcet 

15 control pop-up) . 

Math's and Event Steps 

The math's step allows the definition of 
variables and expressions that set the values of these 
variables. These variables are either numeric or 
20 Boolean and can be set to some mathematical function of 
other variables or properties of ActiveX controls or 
some combination of these. (See Figure 7). 

An event step allows an event to be fired into 
Visual Basic. Thus some Visual Basic can get control 
25 during a sequence to perform some predefined functions. 

Hierarchical Menu Module As Related To Machine Vision 
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S ystem Overview 

The Interactive Graf cetActiveX environment 

System of the present invention uses image analysis 

techniques to perform image analysis and metrology 

5 functions, i.e. calibrated measurement. Features such 

as lines, circles and points etc. can be found and 

measured. Complex shapes can be identified and veri- 
fied. 



10 Complete complex applications can be config- 

ured literally in minutes, and controlled remotely - 
using either a digital or a serial link. 

The features of the system include : 

• Ease of Use 

15 • Calibrated Measurement Using Real World 

Units 

• Extensive Set of Inspection Commands 
Efficient Use of Memory 

Simple Digital and Serial Interfacing 

2 0 There are two phases in the operation of the 

system of the present invention: 

Teach Phase: Here the user, using the 
computer keyboard and monitor, configures 
the system for his application e.g. part 

25 identification or measurement. 

• Execution Phase: Once configured, the 
teach system can run completely automati- 
cally and can be controlled via an RS232 
serial link or it's digital inputs. 



30 



In the first phase the user "teaches" the 
system do a particular application. Using the computer 
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keyboard 12 and monitor 10, the user "configures" the 
system for a particular application e.g. part identifi- 
cation, measurement etc. Use of the system is made easy 
by the menu-driven user interface which provides the 
5 user with all the commands in a structured and hierar- 
chical manner. Help is available at all points in the 
program . 

The system stores the particular sequence of 
commands that make the application into an application 
10 file. More than one application can be loaded at any 
one time. 

In the second phase, applications can be 
executed at the press of a button on the keyboard, a 
remote input, or triggered through the RS232 or other 
15 communications line. Automatic system diagnostics 
provide details of the performance of the application to 
the user, showing the nature and type of any problems. 

All measurements are made in real world co- 
ordinates (m , mm, microns, in., thou., etc.). 

20 Many commands are available to include metrol- 

ogy (measurement) functions and image analysis func- 
tions. Examples of metrology functions include the 
measurement of distances, angles, diameters, perimeter, 
areas, straightness , roundness and shape. Examples of 

25 vision functions include gray level edge finding (capa- 
ble of finding subtle grey level edges) , projection, 
edge sniffing and feature correlation. 

System functions can be divided into 3 main 
categories : 
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• Image Analysis Functions: These func- 
tions determine features of an image 
System Calibration: These functions 
allow the system to measure and work in 

5 a user defined co-ordinate frame 

• System I/O: These functions allow the 
system to respond to digital inputs, set 
digital outputs, communicate through a 
serial line, print information and save 

10 and retrieve information from files. 

Image Analysis Functions 

The image analysis functions can be sorted 
into six main categories: 

• Part locate 

15 Part locate identifies a point and 

orientation on the image. All 
search regions, e.g. lines, windows 
and ellipses that are defined after 
a part has been located are defined 

20 relative to this new part frame 

• Geometry definition 

These functions concern the defini- 
tion of the entities that the system 
uses during part inspection. Lines, 
25 ellipses, points and profiles can be 

defined using these functions. 

• Metrology 

These functions concern the measure- 
ment of various quantities between 
3 0 entities defined within the system. 

Distances, areas, perimeters and 
angles can be measured. Some quan- 
tities can be measured between enti- 
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ties of differing types, e.g. dis- 
tance from a point to a line. Geo- 
metric errors can also be measured. 
These include the straightness of a 
line and roundness of a circle. 
Morphology Measurement 

These functions concern the measure- 
ment of surface details, such as 
cracks, in the image. 
Feature Correlation 

This function finds a pre- taught, 
feature (model) , to pixel accuracy 
within a grabbed image . The system 
calculates a "score" that shows the 
level of similarity between- the 
model and the feature in the current 
image. The score is defined as a 
percentage, with 100% being a per- 
fect match. 

2 0 • Morphology enhancement 

These functions concern the enhance- 
ment of surface details, such as 
cracks, within the image. 
Color Image Processing 

25 Calibration 

The purpose of system recalibration is to tell 
the vision system the units it is working in, i.e. mm, 
inches etc. Calibration defines the attitude of the 
camera 24 relative to the field of view. Camera cali- 

3 0 brat ion is accomplished by viewing a object of known 

shape and size, usually a square or round plate. 
Calibration files are saved with the job file thus 



10 
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allowing different jobs to have different calibration 
data . 

System calibration can be implemented using 
several methods. The first method/phase has been 
5 implemented using several methods. The full methods 
include : 

• Initial System Camera Calibration: This 

occurs whenever the camera 24 is moved, 
or every 6 months on system maintenance. 
10 At calibration time an object of known 

size is placed in the field of view of 
each camera 24, its size and location 
being defined and identified by the 
system. 

15 • Monitoring Camera Movement and Automatic 

System Recalibration : This is the pro- 
cess whereby the system monitors camera 
movement, and compensates for such move- 
ment. The amount of movement observed is 

20 calculated, the calibration matrices 

adjusted and, if excessive, a warning is 
raised. 

• Continuous Automatic System Recalibra- 

tion: In certain instances system cali- 

2 5 brat ion can be accomplished completely 

automatically. The reference object (s) 
is/are continuously in, or placed in, the 
field of view. The system, therefore, 
calibrates itself and returns information 

3 0 on camera movement. If the camera move- 

ment is so excessive that the reference 
object cannot be seen then a warning is 
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raised. The system continues to operate 
using the last default calibration. 

Menus 



This set of "menus allows different geometric 
5 entities (points, lines, circles, ellipses and profiles) 
to be defined and selected. Such entities can either be 
defined using analysis of the video source or can be 
constructed using existing geometric entities. 

Geometry Menu 



10 Point Define Menu 

Point on Edge (Window Scan) 
Point Define via Feature Location 
Center of an Ellipse 
Line + Line Intersect 

15 Point on a Profile 

Inflection Point 
Deflection Point 

Line Define Menu 

Define by Scanning for Edge 
20 Define by Hand 

Define Via Two Points 
Profile Primary Vector 

Circle Define Menu 

Define by Scanning for an Edge 
2 5 Manual 

Profile Define Menu 



Metrology Menus 



WO 99/14646 



PCT/US98/17066 



-30- 

The metrology menus allow the user to perform 
measurements and implement tests based on the measure- 
ment results. The measurements and test results can be 
saved. Statistical Process Control (SPC) may be per- 
5 formed on the measured results. Metrology functions 
select geometric entries independent of where they are 
defined. When selecting an entity all geometry's are 
displayed, from which to select the required entity. 

Linear Dimensions 

Distance Line to Line 
Distance Line to Point 
Distance Point to Point Menu 
Caliper: Single Pair 
Caliper: Multiple Pairs 
Actual Distance 
Component D i s t anc e 
Circle/Ellipse Radius 

Angular Dimensions 

Angle Line to Line 
2 0 Angle Line 



10 



15 



Area Measurement 



Morphology Measurement 

Intensity Evaluation 

2 5 Connectivity Evaluation 

Geometric Errors Menu 
Parallelism 
Squareness 

Form Error Straightness 

3 0 Form Error Roundness 
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Data Interface 



Several optional interfaces are suggested. 
These interface are NOT compulsory as the interactive 
Graf cet/ActiveX environment can work without them. 
5 However, if any of these interface are present this 
extends the functionality to the Graf cet/ActiveX envi- 
ronment . 

These interface provides a set of defined 
methods and attributes obeying the COM standard. 
10 Display Methods (iDisplay) 

Draw Draws the tool 

ShowDisplayGraphicssplay result graphics 



15 



HideDisplayGraphics 
EnableAutoRef resh 



Hide result graphics 
Enables the tool to 
refresh the result 
display upon use 



20 



DisableAutoRef resh 



Disables the tool 
from refreshing the 
result display upon 
use 



IsAutoRef reshEnabled Returns if the 

tool 1 s auto refresh 
is enabled or dis- 
25 abled 

Information Methods ( ilnf ormation) 



GetMemorySize 



Returns the current 
quantity of heap, in 
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SetMaximumTime 



bytes, which the 
tool is using 
Sets the maximum 
time allows for the 
use function to exe- 
cute 



GetMinimumTime 



10 



Returns the minimum 
time required for 
the use function to 
execute 



TimeTaken 



Returns the time 
taken by the Use 
function on the last 
use 



15 



GetMaximumTime 



Returns the current 
setting for Use max- 
imum time allowed 
for the function to 
execute 



20 Data Access Methods (iData) 



GetlnputType 



Returns information 
regarding an input 



GetOutput Type 



Returns information 
regarding an output 



25 



Ge t Number 0 f I npu t s 



Returns the number 
of inputs 
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GetNumberOf Outputs 

GetOutput 

5 Set Input 

Tool Methods (iTool) 
AutoTrain 

Train 

10 

Adds ample 



PrepareToUse 

15 

Q 

Use 



GetLastError 

20 

Save 



Returns the number 
of outputs 

Returns a given out- 
put 

Set a given input 



Automatic Training 
method for the tool 

Initiate training of 
the tool 

Add Sample to cur- 
rent training set 
for the tool 

Prepares the tool 
for first use 

Uses the tool with 
the current trained 
information 

Returns the last 
error generated by 
the tool 

Save tool configura- 
tion and training 
information to file 
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Load 



Load tool configura- 
tion and training 
information to file 



Serialize 



Serializes the tool 
configuration to 
file 



While the best mode for carrying out the 
invention has been described in detail, those familiar 
with the art to which this invention relates will 
10 recognize various alternative designs and embodiments 
for practicing the invention as defined by the following 
claims . 
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What Is Claimed Is : 

1. . A method for interactively developing a 
graphical, control-flow structure and associated appli- 
cation software for use in a machine vision system 
5 utilizing a computer system, the structure including a 
control sequence having at least one node, and at least 
one transition, and at least one link for interconnect- 
ing the at least one node and the at least one transi- 
tion, the method comprising the steps of: 

10 storing an application development program, 

including a first set of control programs representing 
possible components of a user interface for the machine 
vision system, the first set of control programs defin- 
ing a first set of standard controls; 

15 storing a second set of control programs 

representing possible machine vision algorithms for the 
machine vision system, the second set of control pro- 
grams defining a second set of standard controls; 

' storing hardware operating parameters corre- 

2 0 sponding to possible hardware for use in the machine 

vision system, the hardware operating parameters defin- 
ing a third set of standard controls; 

displaying graphical representations of the 
possible components, the possible hardware and the 
25 possible machine vision algorithms; 

receiving commands from a user of the computer 
system to select a first control program corresponding 
to a desired component of the user interface, desired 
hardware operating parameters corresponding to desired 

3 0 hardware and a machine vision graphical representation 

and its associated second control program corresponding 
to a desired machine vision algorithm; 

displaying the structure wherein the machine 
vision graphical representation is a node of the struc- 
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ture, the selected second control program being mapped 
into the structure at the node, the displayed structure 
also having at least one transition and at least one 
link for interconnecting the node and the at least one 
transition; and 

linking the first control program with the 
desired hardware operating parameters to the second 
control programs to form the application software in 
response to the commands without the user writing any of 
the application software. 

2 . The method as claimed in claim 1 wherein 
the standard controls include properties and wherein the 
step of linking includes the step of setting the proper- 
ties of the standard controls. 

3 . The method as claimed in claim 2 wherein 
the step of setting includes the step of setting one 
property of one standard control to be equal to one 
property of another standard control . 

4 . The method as claimed in claim 1 wherein 
the structure has a hierarchy with a top and a bottom 
and wherein the hierarchy allows bottom-up or top-down 
development . 

5. The method as claimed in claim 1 further 
comprising the steps of: 

storing a third set of control programs repre- 
senting possible software products for use in the 
machine vision system; 

displaying machine vision graphical repre- 
sentations of the possible software products; and 

displaying the graphical representation of a 
selected software product in the structure as another 
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node or an input /output to the node of the structure, 
wherein the step of linking links a third control 
program corresponding to a desired software product with 
the first and second control programs and the desired 
5 hardware operating parameters in response to the com- 
mands to form the application software. 

6 . The method as claimed in claim 1 wherein 
the desired hardware operating parameters correspond to 
a desired image source of the machine vision system. 

10 

7 . The method as claimed in claim 6 wherein 
the desired image source is a video camera. 

8 . The method as claimed in claim 6 wherein 
the desired operating parameters further correspond to 

15 a desired vision processor board of the machine vision 
system . 

9. The method as claimed in claim 6 wherein 
the desired hardware operating parameters further 
correspond to a desired frame grabber board of the 

2 0 machine vision system. 

10. The method as claimed in claim 6 wherein 
the desired hardware operating parameters further 
correspond to a desired motion board of the machine 
vision system. 

25 11. The method as claimed in claim 6 wherein 

the desired hardware operating parameters further 
correspond to a desired bus of the machine vision 
system . 
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12. The method as claimed in claim 6 wherein 
• the desired machine vision algorithm is an image pro- 
j cessing algorithm. 

13. The method as claimed in claim 6 wherein 
5 the desired machine vision algorithm is an image -analy- 
sis algorithm. 

14. The method as claimed in claim 6 wherein 
the desired machine vision algorithm is a calibration 
space algorithm. 

10 15. The method as claimed in claim 6 wherein 

the desired machine vision algorithm is an interactive 
CAD/geometry algorithm. 

16 . The method as claimed in claim 6 wherein 
the computer system includes a personal computer. 

15 17. A computer system for interactively 

developing a graphical, control-flow structure and 
associated application software for use in a machine 
vision system, the structure including a control se- 
quence having at least one node, and at least, one 

2 0 transition, and at least one link for interconnecting 
the at least one node and the at least one transition, 
the computer system comprising: 

means for storing an application development 
program, including a first set of control programs 

25 representing possible components of a user interface for 
the machine vision system, the first set of control 
programs defining a first set of standard controls; 

means for storing a second set of control 
programs representing possible machine vision algorithms 
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for the machine vision system, the second set of control 
programs defining a second set of standard controls; 

means for storing hardware operating parame- 
ters corresponding to possible hardware for use in the 
5 machine vision system, the hardware operating parameters 
defining a third set of custom controls; 

a display for displaying graphical representa- 
tions of the possible components, the possible hardware 
and the possible machine vision algorithms; 

10 means for receiving commands from a user of 

the computer system to select a first control program 
corresponding to a desired component of the user inter- 
face, desired hardware operating parameters correspond- 
ing to desired hardware and a machine vision graphical 

15 representation and its associated second control program 
corresponding to a desired machine vision algorithm, the 
display also displaying the structure wherein the 
machine vision graphical representation is a node of the 
structure, the selected second control program being 

2 0 mapped into the structure at the node, the displayed 

structure also having at least one transition and at 
least one link for interconnecting the node and the at 
least one transition; and 

means for linking the first control program 
25 with the desired hardware operating parameters to the 
second control program to form the application software 
in response, to the commands without the user writing any 
of the application software. 

18. The computer system as claimed in claim 

3 0 17 wherein the standard controls include properties and 

wherein the means for linking includes means for setting 
the properties of the standard controls. 
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19. The computer system as claimed in claim 
18 wherein the means for setting sets one property of 
one standard control to be equal to one property of 
another standard control . 

5 20. The computer system as claimed in claim 

17 wherein the structure has a hierarchy with a top and 
a bottom and wherein the hierarchy allows bottom-up or 
top-down development . 

10 21. The computer system as claimed in claim 

17 further comprising: 

means for storing a third set of control 
programs representing possible software products for use 
in the machine vision system, the display displaying 

15 graphical representations of the possible software 
products and a graphical representation of a selected 
software product in the structure as another node or an 
input /output to the node of the structure, wherein the 
step of linking links a third control program corre- 

20 sponding to a desired software product with the first 

and second control programs and the desired hardware 
operating parameters in response to the commands to form 
the application software, 

22 . The computer system as claimed in claim 
25 17 wherein the desired hardware operating parameters 

correspond to a desired image source of the machine 
vision system . 

23 . The computer system as claimed in claim 
22 wherein the desired image source is a video camera. 

3 0 24. The computer system as claimed in claim 

22 wherein the desired operating parameters further 
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correspond to a desired vision processor board of the 
machine vision system. 

25. The computer system as claimed in claim 
22 wherein the desired hardware operating parameters 

5 further correspond to a desired frame grabber board of 
the machine vision system. 

26. The computer system as claimed in claim 
22 wherein the desired hardware operating parameters 
further correspond to a desired motion board of the 

10 machine vision system. 

27. The computer system as claimed in claim 
22 wherein the desired hardware operating parameters 
further correspond to a desired bus of the machine 
vision system. 

15 28. The computer system as claimed in claim 

22 wherein the desired machine vision algorithm is an 
image processing algorithm. 

29. The computer system as claimed in claim 
22 wherein the desired machine vision algorithm is an 

20 image-analysis algorithm. 

30. The computer system as claimed in claim 
22 wherein the desired machine vision algorithm is a 
calibration space algorithm. 

31. The computer system as claimed in claim 
25 22 wherein the desired machine vision algorithm is an 

interactive CAD/geometry algorithm. 
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32. The computer system as claimed in claim 
22 wherein the computer system includes a personal 
computer . 
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